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Abstract 


The  End-To-End  Test  (ETE)  is  being  conducted  under  the  auspices  of  the  Joint  Advanced 
Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  (JT&E).  The  purpose  of  the  ETE  is  to 
investigate  the  utility  of  using  advanced  distributed  simulation  (ADS)  to  augment  both 
developmental  and  operational  testing  of  the  Joint  Surveillance  Target  Attack  Radar  System 
(STARS).  The  basic  concept  behind  the  ETE  is  to  augment  the  Joint  STARS  environment  with  a 
virtual  environment  created  by  thousands  of  simulated  entities  or  targets.  This  virtual 
environment  is  imaged  by  a  simulation  of  the  radar  system  onboard  the  Joint  STARS  E-8C 
aircraft. 

The  simulation  environment  representing  the  radar  subsystem  onboard  the  E-8C  aircraft  is  called 
the  Virtual  Surveillance  Target  Attack  Radar  System  (VSTARS).  VSTARS  uses  a  combination 
of  actual  radar  subsystem  software,  simulations  of  aircraft  subsystems,  and  two  near  real-time 
stochastic  radar  simulations  with  necessary  databases  and  libraries  to  simulate  the  Joint  STARS 
radar  subsystem. 

VSTARS  uses  early  engineering  models  and  simulations  and  results  of  early  developmental  tests 
to  develop  a  simulation  that  may  be  used  to  assess  cost-performance  trade-off,  technical  risk, 
system  maturity,  operational  effectiveness,  suitability,  training,  mission  planning,  and  employment. 

The  adaptation  of  VSTARS  as  the  primary  training  simulation  for  Joint  STARS  by  the  Air  Force 
represents  the  last  step  (pun  intended)  in  the  application  of  the  simulation,  test  and  evaluation 
process  (STEP)  to  Joint  STARS  and  establishes  the  validity  of  the  process  in  support  of  the 
Department  of  Defense  acquisition  process. 

Background 

The  Joint  Advanced  Distributed  Simulation  Joint  Test  and  Evaluation  was  chartered  by  the  Office 
of  the  Under  Secretary  of  Defense  (Acquisition  and  Technology),  Office  of  the  Director,  Test  and 
Evaluation  “...to  investigate  the  utility  of  Advanced  Distributed  Simulation  (ADS)  for  both 
developmental  test  and  evaluation  (DT&E)  and  operational  test  and  evaluation  (OT&E).  JADS 
will  investigate  the  present  utility  of  ADS,  including  Distributed  Interactive  Simulation,  for  test 
and  evaluation  (T&E);  and  finally,  identify  the  requirements  that  must  be  introduced  in  ADS 
systems  if  they  are  to  support  a  more  complete  T&E  capability  in  the  future.” 
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The  End-To-End  Test  is  one  of  three  tests  within  the  JADS  JT&E  program.  The  ETE  is  designed 
to  evaluate  the  utility  of  ADS  to  support  testing  of  a  command,  control,  communications, 
computers,  intelligence,  surveillance  and  reconnaissance  (C4ISR)  system,  the  Joint  STARS,  while 
the  system  performs  the  end-to-end  loop  of  detect-track-target-cue  a  weapons  system  and 
assesses  battle  damage.  Joint  STARS  is  composed  of  both  an  airborne  and  ground  segment  along 
with  the  necessary  communications  subsystems.  The  E-8  airborne  system  and  the  ground  station 
modules  (GSM)  together  provide  the  surveillance,  target  detection,  and  tracking  required  to  assist 
commanders  in  understanding  the  enemy  situation  and  taking  action  to  destroy  enemy  forces. 

During  the  feasibility  study  phase  of  JADS,  it  quickly  became  apparent  that  a  simulation  or 
simulations  of  the  radar  subsystem  that  could  provide  the  required  fidelity,  operate  in  real  time 
throughout  the  ground  radar  coverage  area,  and  emulate  the  two  radar  modes,  moving  target 
indicator  (MTI)  and  synthetic  aperture  radar  (SAR)  did  not  exist.  Most  MTI  simulations 
represented,  at  best,  a  single  operator's  workstation,  used  estimated  timelines,  and  performed  as  if 
they  had  a  dedicated  radar  subsystem.  A  feasible  SAR  simulation  did  not  exist.  Common  practice 
was  to  substitute  preloaded  images  that  often  had  little  or  no  operational  relevance. 

If  such  a  simulation  could  be  developed,  it  could  be  used  with  existing  simulations,  such  as  the 
navigation  subsystem  simulation,  to  emulate  the  performance  of  the  E-8  airborne  radar  subsystem. 
By  adding  additional  subsystem  simulations,  hardware  components,  software,  or  fielded 
components,  the  entire  Joint  STARS  could  be  represented  in  configurations  ranging  from  pure 
simulation  to  an  ADS-augmented  fielded  system. 

As  a  result  of  the  decision  by  JADS  to  develop  such  a  simulation,  the  Virtual  Surveillance  Target 
Attack  Radar  System  was  designed  by  JADS  and  built  by  Northrop  Grumman,  Lockheed  Martin, 
and  Motorola.  VSTARS  is  capable  of  functioning  in  a  pure  laboratory  environment  using 
commercial  off-the-shelf  equipment,  in  a  hardware-in-the-loop  environment  using  fielded 
components  (primary  mission  equipment),  and  onboard  the  aircraft  coexisting  with  the  real  radar 
subsystem  in  an  ADS-augmented  environment. 

VSTARS  uses  early  engineering  models  and  simulations,  the  results  of  early  developmental  tests, 
specially  developed  radar  emulations,  and  Joint  STARS  equipment  and  software  to  represent  the 
E-8C  subsystem.  When  connected  to  a  GSM,  simulated,  physical  prototype,  or  real,  it  represents 
the  entire  system. 

Since  the  purpose  of  this  paper  is  to  talk  about  STEP,  I  will  say  no  more  about  VSTARS  and  its 
development  other  than  it  has  undergone  a  rigorous  verification  and  validation  (V&V)  to  include 
a  modified  Turing  test.  The  V&V  has  been  reviewed  by  a  multiservice  accreditation  board,  and  it 
has  been  accredited  for  use  in  the  ETE.  In  addition,  VSTARS  is  currently  undergoing  a  radar 
acceptance  test  conducted  by  the  joint  program  office’s  joint  test  force  similar  to  that  conducted 
prior  to  acceptance  of  a  production  radar  aircraft. 

VSTARS  and  STEP 


2 


How  than  does  STEP  apply  to  VSTARS,  and  why  would  we  consider  it  a  STEP  success  story? 
After  all,  Joint  STARS  is  a  fielded  system,  and  VSTARS  was  developed  for  a  joint  test.  This 
question  is  best  answered  by  reviewing  the  definition  of  STEP  as  contained  in  the  Simulation, 
Test  and  Evaluation  Process  (STEP)  Guidelines,  4  Dec  97. 

“The  Simulation  Test  and  Evaluation  Process  is  defined  as  an  iterative  process  that 
integrates  simulation  and  test  for  the  purpose  of  interactively  evaluating  and 
improving  the  design,  performance,  joint  military  worth,  survivability,  suitability, 
and  effectiveness  of  systems  to  be  acquired  and  improving  how  those  systems  are 
used.” 

The  key  phrase  for  us  to  consider  in  this  definition  is  that  STEP  is  defined  as  an  iterative  process. 
Joint  STARS  is  a  fielded  system,  true,  but  it  is  hopefully  only  at  the  beginning  of  its  life  cycle.  If 
the  examples  set  by  similar  systems,  such  as  the  Airborne  Warning  and  Control  System,  are  valid, 
it  will  undergo  many  more  iterations  before  being  retired.  Therefore,  if  VSTARS  can  be 
successfully  integrated  with  testing  for  the  purpose  of  interactively  evaluating  and  improving  the 
design,  performance,  joint  military  worth,  survivability,  suitability,  and  effectiveness  of  Joint 
STARS,  then  VSTARS  is  a  STEP  success  story. 

Joint  STARS  -  Almost  STEP 


Prior  to  describing  how  VSTARS  can  be  integrated  into  the  testing  and  acquisition  process  for 
Joint  STARS,  it  is  first  necessary  to  review  the  program  and  its  applicability  to  STEP.  As  stated, 
Joint  STARS  is  a  fielded  system  that  existed  prior  to  Dr.  Paul  Kaminski’s  announcement  requiring 
STEP  to  be  a  part  of  future  test  and  evaluation  master  plans.  A  review  of  the  Joint  STARS 
program,  however,  reveals  that  many  of  the  practices,  and  even  more  importantly,  the  resources 
needed  by  STEP  are  present  and  in  use  by  the  developers  and  testers  of  the  system.  There  exists 
today  an  extensive  set  of  laboratory  facilities,  component  test  facilities,  subsystem  simulators, 
hardware-in-the-loop  facilities,  engineering  models,  and  subsystem  simulations  that  are  used  to 
support  the  development  and  testing  of  Joint  STARS. 

What  was  missing  was  a  system  representation  which  due  to  the  distributed  nature  of  the  system 
would  require  a  distributed  simulation.  This  system  representation  would  provide  the  ability  to 
evaluate  the  design,  performance,  joint  military  worth,  survivability,  and  effectiveness  of  Joint 
STARS  as  a  whole.  As  an  aside,  up  to  now  this  evaluation  or  testing  has  primarily  required 
physical  prototypes,  test  aircraft,  and  costly  field  trials. 

The  Success  Story 

We  at  JADS  consider  VSTARS  to  be  that  missing  system  representation.  To  validate  that  belief, 
I  would  like  to  describe  how  VSTARS  can  be  integrated  with  testing  for  the  purpose  of 
interactively  evaluating  and  improving  the  design,  performance,  joint  military  worth,  survivability, 
suitability,  and  effectiveness  of  Joint  STARS.  In  addition,  I  will  describe  how  VSTARS  can,  and 
most  likely  will,  be  used  in  the  development  of  training  simulators  with  the  requisite  realism  and 
fidelity  for  system  operators  and  maintainers. 
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Developmental  Testing 


VSTARS  was  designed  so  that  the  only  true  simulation  is  the  simulation  of  the  MTI  and  SAR 
radar  modes  taking  place  within  the  radar  subsystem.  Everything  else  is  either  integration  code  or 
actual  E-8C  system  code.  The  inputs  into  VSTARS,  except  for  the  target  data,  are  the  inputs  into 
the  real  radar  processor,  and  the  outputs  are  the  actual  radar  reports.  The  radar  simulations  are 
parallel  processes  with  the  radar  when  live  and  virtual  are  mixed  and  solve  the  radar  equations  in 
order  to  achieve  the  required  fidelity. 

This  architecture  allows  the  use  of  VSTARS  to  conduct  developmental  testing  of  all  of  the  other 
subsystems  that  comprise  Joint  STARS,  provided  VSTARS  is  an  accurate  representation  of  the 
radar.  Obviously,  VSTARS  cannot  be  used  to  conduct  developmental  testing  of  the  radar 
subsystem. 

As  an  example,  one  of  the  features  of  the  workstation  used  on  the  E-8C  is  an  automatic  tracker 
(A-tracker).  The  A-tracker  works  off  radar  reports  and  when  initiated  will  automatically  track  a 
designated  formation  providing  bearing,  speed,  and  number  of  vehicles.  Prior  to  VSTARS,  it  was 
necessary  to  either  have  a  functioning  radar  (test  flight)  or  a  recording  of  a  functioning  radar  in 
order  to  test  the  A-tracker.  Test  cases  were  basically  limited  to  those  that  could  be  achieved  at 
Eglin  Air  Force  Base,  Florida,  with  a  minimal  number  of  vehicles  traveling  under  peacetime  safety 
restrictions. 

Northrop  Grumman  is  currently  developing  an  annual  release  of  the  radar  software  that  will 
incorporate  a  revised  version  of  the  A-tracker  software.  Parallel  to  the  normal  testing  of  the  new 
A-tracker  software,  Northrop  Grumman  also  conducted  a  study  to  determine  if  VSTARS  could 
be  used  to  test  this  software.  They  found  that  both  the  old  and  new  versions  of  the  A-tracker 
software  functioned  as  expected  with  VSTARS.  The  old  version  also  functioned  the  same  as  it 
did  using  the  live  radar,  which  was  to  be  expected  as  the  radar  reports  it  uses  are  identical. 

The  findings  from  this  study  were  encouraging  and  verify  some  of  the  premises  of  STEP.  Test 
cases  could  be  “flown”  using  VSTARS  whenever  needed  with  as  many  repetitions  as  desired.  This 
was  possible  without  competing  for  scarce  test  aircraft  and  range  resources.  Cost  was  a  “no 
brainer.”  One  tester  and  two  computers  in  a  lab  vice  the  aircraft,  testers,  crew,  and  range  assets 
required  for  a  live  test.  Any  conceivable  test  case  could  be  “flown”  without  worrying  about  safety 
or  limited  assets.  Bad  solutions  could  be  quickly  discarded,  and  new,  better  solutions  could  be 
tried  the  next  day.  Most  importantly,  when  a  live  test  flight  is  flown,  as  it  must  be,  the  testers  can 
be  reasonably  sure  that  they  will  get  the  maximum  value  from  the  flight  and  test  conditions. 

Operational  Testing 

Operational  testing  (OT),  early  operational  assessments,  and  the  development  of  tactics, 
techniques  and  procedures  are  all  areas  where  STEP  will  make  a  large  impact.  As  systems 
become  larger  and  more  complex,  it  becomes  more  and  more  difficult  to  effectively  test  them. 
This  was  the  case  with  Joint  STARS,  a  corps-  or  theater-level  C4ISR  system  that  was  scheduled 
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to  be  tested  with  a  brigade-level  force.  Operational  testing  of  a  C4ISR  system,  specifically  Joint 
STARS,  has  been  the  main  thrust  of  the  JADS  End-To-End  Test. 

To  date,  JADS  ETE  has  conducted  a  laboratory-based  operational  test  of  Joint  STARS  and  an 
operational  assessment  of  the  ability  of  Joint  STARS  to  interact  with  the  Army’s  All  Source 
Analysis  System  (ASAS).  The  Joint  STARS  radar  information  was  used,  along  with  other 
intelligence  sources,  to  determine  targets  for  engagement  by  a  virtual  Army  Tactical  Missile 
System  (ATACMS).  Additional  details  may  be  found  in  another  paper  presented  at  these 
proceedings  entitled  ETE  Update  -  ADS  Testing  of  C4ISR  Systems. 

By  using  ADS  and  VSTARS,  JADS  ETE  was  able  to  place  a  simulated  Joint  STARS  in  a  realistic 
operational  environment  and  collect  valid  data  for  the  evaluation  of  operational  measures. 
Additionally,  though  not  early,  operational  assessments  were  made  as  to  the  ability  of  Joint 
STARS  to  accurately  detect  and  report  on  potential  ATACMS  targets.  Finally,  numerous  tactics, 
techniques  and  procedures  were  developed  covering  the  interaction  of  Joint  STARS  and  ASAS 
during  the  training  and  rehearsal  periods  of  the  test. 

In  the  March  timeframe,  JADS  ETE  will  again  conduct  an  operational  test  of  Joint  STARS  and  an 
operational  assessment  of  the  ability  of  Joint  STARS  to  interact  with  the  Army’s  All  Source 
Analysis  System.  The  major  difference  will  be  the  use  of  an  actual  ADS-augmented 
E-8C  for  two  of  the  test  days.  Joint  STARS  will  see  both  real  and  virtual  targets  under  various 
clutter  conditions  while  operating  in  a  realistic  operational  environment  and  collecting  data  for  the 
evaluation  of  operational  measures. 

Training 

The  Simulation,  Test  and  Evaluation  Process  (STEP)  Guidelines  state  that  STEP  can  aid  in  the 
development  of  training  simulators  with  the  requisite  realism  and  fidelity  for  system  operators  and 
maintainers.  Such  is  the  case  with  VSTARS.  It  is  currently  being  integrated  into  the  current 
Mission  Crew  Training  System  as  a  prototype  system  and,  if  successful,  will  likely  be  the  new 
trainer  for  the  Air  Force. 

The  Future 


As  stated  earlier,  STEP  is  applicable  for  the  entire  life  cycle  of  the  system.  Like  all  systems,  Joint 
STARS  is  constantly  undergoing  improvement  and  upgrades.  Inquiries  have  been  made  as  to  the 
possibility  of  using  VSTARS  for  the  Joint  STARS  Block  2  follow-on  OT&E.  In  addition,  the 
Joint  STARS  Joint  Test  Force  is  currently  extending  the  V&V  of  VSTARS  so  that,  if  within 
specifications,  VSTARS  may  be  accredited  for  use  in  the  testing  of  subsystems  such  as  the  A- 
tracker  or  a  modified  operator  workstation.  Also,  if  the  decision  is  made  to  retest  the  common 
ground  station  (CGS),  VSTARS  could  be  used  to  solve  or  answer  many  of  the  faults  found  in  the 
first  test.  For  example,  operators  can  easily  be  trained  on  the  new  CGS,  and  joint  tactics, 
techniques  and  procedures  can  be  developed  and  practiced  using  a  synthetic  environment  similar 
to  that  used  by  the  JADS  End-to-End  Test. 
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Northrop  Grumman  is  also  interested  in  using  appropriately  modified  versions  of  VSTARS  to 
provide  early  operational  assessments  of  proposed  product  improvements  to  the  radar  used  by 
Joint  STARS.  Also,  if  the  product  improvements  are  authorized,  VSTARS  could  then  be  used  to 
(1)  verify  the  system’s  design;  (2)  confirm  that  design  risks  have  been  controlled;  (3)  certify 
readiness  for  operational  testing;  and  (4)  evaluate  the  system’s  operational  effectiveness, 
suitability,  and  survivability. 

Conclusions 


Last  year  at  the  ITEA  conference  in  Orlando,  Florida,  Dr.  Patricia  Sanders,  Director,  Test, 
Systems  Engineering  and  Evaluation  stated  the  implications  of  STEP  to  the  acquisition 
community. 

•  Testers  will  be  involved  much  earlier  in  an  acquisition  program. 

•  Predictions  of  system  performance  can  be  made  to  assess  the  military  worth  of  the 
system  before  any  physical  prototypes  are  built. 

•  Predictive  simulations  can  be  used  to  expand  the  scope  of  system  testing  and  evaluate 
system  performance  in  areas  that  are  not  readily  testable. 

•  The  models,  simulations,  and  test  data  will  have  a  much  wider  use  throughout  the 
system’s  life  cycle. 

The  implications  of  VSTARS  to  the  Joint  STARS  acquisition  community  are  the  same. 

•  Both  DT  and  OT  testers  will  be  involved  much  earlier  in  the  product  improvement 
program. 

•  Predictions  of  system  performance  will  be  made  to  assess  the  military  worth  of  the 
system  before  any  physical  prototypes  are  built. 

•  Predictive  simulations  will  be  used  to  expand  the  scope  of  system  testing  and  evaluate 
system  performance  in  areas  that  are  not  readily  testable. 

•  The  models,  simulations,  and  test  data  will  have  a  much  wider  use  throughout  the 
remainder  of  the  system’s  life  cycle. 
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